Flower Image Classifier

Project

Machine learning is getting incorporated more and more into everyday apps. Soon, anything with a camera will use image recognition which is based on deep learning models. The goal of the project is to build such a model for the purpose of recognizing flower species and use it in a simple application. The app will recognize flower types that are on the images.

Import resources

In [1]:
import warnings
warnings.filterwarnings('ignore')
In [2]:
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import json
import numpy as np
import matplotlib.pyplot as plt

import tensorflow as tf
import tensorflow_datasets as tfds
import tensorflow_hub as hub
tfds.disable_progress_bar()
In [3]:
import logging
logger = tf.get_logger()
logger.setLevel(logging.ERROR)
In [4]:
# The new version of dataset is only available in the tfds-nightly package
%pip --no-cache-dir install tfds-nightly --user
# need to restart the kernel after it is run for the first time
Requirement already satisfied: tfds-nightly in /home/adam/.local/lib/python3.8/site-packages (3.1.0.dev202006230105)
Requirement already satisfied: requests>=2.19.0 in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from tfds-nightly) (2.24.0)
Requirement already satisfied: absl-py in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from tfds-nightly) (0.9.0)
Requirement already satisfied: tqdm in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from tfds-nightly) (4.46.1)
Requirement already satisfied: promise in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from tfds-nightly) (2.2.1)
Requirement already satisfied: protobuf>=3.6.1 in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from tfds-nightly) (3.12.3)
Requirement already satisfied: tensorflow-metadata in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from tfds-nightly) (0.14.0)
Requirement already satisfied: future in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from tfds-nightly) (0.18.2)
Requirement already satisfied: termcolor in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from tfds-nightly) (1.1.0)
Requirement already satisfied: dill in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from tfds-nightly) (0.3.1.1)
Requirement already satisfied: six in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from tfds-nightly) (1.15.0)
Requirement already satisfied: wrapt in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from tfds-nightly) (1.12.1)
Requirement already satisfied: attrs>=18.1.0 in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from tfds-nightly) (19.3.0)
Requirement already satisfied: numpy in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from tfds-nightly) (1.18.5)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from requests>=2.19.0->tfds-nightly) (1.25.9)
Requirement already satisfied: certifi>=2017.4.17 in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from requests>=2.19.0->tfds-nightly) (2020.6.20)
Requirement already satisfied: idna<3,>=2.5 in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from requests>=2.19.0->tfds-nightly) (2.9)
Requirement already satisfied: chardet<4,>=3.0.2 in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from requests>=2.19.0->tfds-nightly) (3.0.4)
Requirement already satisfied: setuptools in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from protobuf>=3.6.1->tfds-nightly) (47.3.1.post20200622)
Requirement already satisfied: googleapis-common-protos in /home/adam/anaconda3/envs/tf_env/lib/python3.8/site-packages (from tensorflow-metadata->tfds-nightly) (1.51.0)
Note: you may need to restart the kernel to use updated packages.

Dataset

The dataset contains images of 102 flower types

Load

In [5]:
%%capture

# Download data to default local directory "~/tensorflow_datasets"
!python -m tensorflow_datasets.scripts.download_and_prepare --register_checksums=True --datasets=oxford_flowers102

# Load the dataset with TensorFlow Datasets
dataset, dataset_info = tfds.load('oxford_flowers102', as_supervised = True, with_info = True)


# Create a training set, a validation set and a test set
training_set = dataset['train']
validation_set = dataset['validation']
test_set = dataset['test']

Explore

In [6]:
# Get the number of examples in each set from the dataset info
print("Number of examples:")
num_training_examples = dataset_info.splits['train'].num_examples
print("- training", num_training_examples)
num_validation_examples = dataset_info.splits['validation'].num_examples
print("- validation", num_validation_examples)
num_test_examples = dataset_info.splits['test'].num_examples
print("- test", num_test_examples)

# Get the number of classes in the dataset from the dataset info
num_classes = dataset_info.features['label'].num_classes
print("\nNumber of classes", num_classes)
Number of examples:
- training 1020
- validation 1020
- test 6149

Number of classes 102
In [7]:
# Print the shape and corresponding label of 3 images in the training set
print("Shape and label of the first 3 images:")
for image, label in training_set.take(3):
    print('\u2022 shape', image.shape, '\n\u0020 label', label.numpy())
Shape and label of the first 3 images:
• shape (500, 667, 3)
  label 72
• shape (500, 666, 3)
  label 84
• shape (670, 500, 3)
  label 70
In [8]:
# Plot 1 image from the training set and its label
for image, label in training_set.take(1):
    plot_title = label.numpy()

    plt.imshow(image, cmap = plt.cm.binary)
    plt.title(plot_title)
    plt.colorbar()
    plt.show()

Label mapping

Load the flower names corresponding to the labels.

In [9]:
with open('label_map.json', 'r') as f:
    class_names = json.load(f)
In [10]:
# Plot the image again, this time with corresponding class name
for image, label in training_set.take(1):
    plot_title = class_names[str(label.numpy())]

    plt.imshow(image, cmap = plt.cm.binary)
    plt.title(plot_title)
    plt.colorbar()
    plt.show()

Create pipeline

In [11]:
BATCH_SIZE = 64
IMG_SHAPE = 224

# Normalize color range of an image from 0-255 to 0-1
def normalize(image, label):
    image = tf.cast(image, tf.float32)
    image /= 255
    return image, label

# Resize image
def resize(image, label):
    image = tf.image.resize(image, (IMG_SHAPE, IMG_SHAPE))
    return image, label

# Apply the transformations and return a pipeline
def batchesOfImages(set):
    return set.cache() \
        .shuffle(num_training_examples // 4) \
        .map(resize) \
        .batch(BATCH_SIZE) \
        .map(normalize) \
        .prefetch(1)

# Create a pipeline for each set
training_batches = batchesOfImages(training_set)
validation_batches = batchesOfImages(validation_set)
test_batches = batchesOfImages(test_set)

Build and train the classifier

The data is ready. Now is the time to build the model. Image classification is a challenging problem, for this reason the model will be built with the use of a pre-trained neural network. This approach is called transfer learning and it involves tuning the existing network that has been trained for a similar task.

This network is MobileNet and it comes from TensorFlow Hub repository. It has been trained on ImageNet, a massive dataset with over 1 million labeled images in 1000 categories. There is a lot of intelligence stored in this model which will be helpful in building the image classifier for flowers.

Because MobileNet is a network trained for a general image recognition, the best approach would be to tune it by just replacing the output layer and training it for classifying flowers. The fact that flowers dataset is small is another argument for keeping the weights of the pre-trained network constant.

Load pre-trained network

It will be responsible for extracting the features of images.

In [12]:
# Load MobileNet
URL = "https://tfhub.dev/google/tf2-preview/mobilenet_v2/feature_vector/4"

feature_extractor = hub.KerasLayer(URL, input_shape=(IMG_SHAPE, IMG_SHAPE, 3))

# Set its weights constant
feature_extractor.trainable = False

Build the model

Add output layer with 102 output nodes.

In [13]:
model = tf.keras.Sequential([
    feature_extractor,
    tf.keras.layers.Dense(102, activation = 'softmax')
])

model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
keras_layer (KerasLayer)     (None, 1280)              2257984
_________________________________________________________________
dense (Dense)                (None, 102)               130662
=================================================================
Total params: 2,388,646
Trainable params: 130,662
Non-trainable params: 2,257,984
_________________________________________________________________

Train the model

In [14]:
# Train the model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

EPOCHS = 10

history = model.fit(training_batches,
                    epochs = EPOCHS,
                    validation_data = validation_batches)
Epoch 1/10
16/16 [==============================] - 21s 1s/step - loss: 4.5332 - accuracy: 0.0696 - val_loss: 3.6489 - val_accuracy: 0.2549
Epoch 2/10
16/16 [==============================] - 19s 1s/step - loss: 2.8518 - accuracy: 0.5069 - val_loss: 2.6367 - val_accuracy: 0.5471
Epoch 3/10
16/16 [==============================] - 20s 1s/step - loss: 1.7946 - accuracy: 0.8000 - val_loss: 2.0153 - val_accuracy: 0.6696
Epoch 4/10
16/16 [==============================] - 22s 1s/step - loss: 1.1610 - accuracy: 0.9167 - val_loss: 1.6534 - val_accuracy: 0.7127
Epoch 5/10
16/16 [==============================] - 20s 1s/step - loss: 0.8021 - accuracy: 0.9539 - val_loss: 1.4282 - val_accuracy: 0.7500
Epoch 6/10
16/16 [==============================] - 20s 1s/step - loss: 0.5866 - accuracy: 0.9765 - val_loss: 1.2886 - val_accuracy: 0.7529
Epoch 7/10
16/16 [==============================] - 19s 1s/step - loss: 0.4488 - accuracy: 0.9863 - val_loss: 1.1924 - val_accuracy: 0.7716
Epoch 8/10
16/16 [==============================] - 19s 1s/step - loss: 0.3545 - accuracy: 0.9922 - val_loss: 1.1213 - val_accuracy: 0.7716
Epoch 9/10
16/16 [==============================] - 20s 1s/step - loss: 0.2859 - accuracy: 0.9931 - val_loss: 1.0656 - val_accuracy: 0.7882
Epoch 10/10
16/16 [==============================] - 19s 1s/step - loss: 0.2358 - accuracy: 0.9961 - val_loss: 1.0253 - val_accuracy: 0.7892

Plot loss and accuracy

In [15]:
epochs = range(1, 11)

plt.subplot(1, 2, 1)
plt.plot(epochs, history.history['loss'], 'b--', label = 'training')
plt.plot(epochs, history.history['val_loss'], 'b', label = 'validation')
plt.legend()
plt.title("Loss in trainig and validation")
plt.xlabel('Epochs')
plt.ylabel('Loss')

plt.subplot(1, 2, 2)
plt.plot(epochs, history.history['accuracy'], 'g--', label = 'training')
plt.plot(epochs, history.history['val_accuracy'], 'g', label = 'validation')
plt.legend()
plt.title("Accuracy in training and validation")
plt.xlabel('Epochs')
plt.ylabel('Accuracy')

plt.tight_layout() # to make plots not overlap
plt.show()

Training result is pretty good at about 80% validation accuracy. Although the distance between the curves grows significantly which indicates overfitting.

Test the model

Real test on a previously unseen data, the testing set. It will give a good estimate of a model's performance in practice.

In [16]:
model.evaluate(test_batches)
97/97 [==============================] - 63s 647ms/step - loss: 1.1335 - accuracy: 0.7642
Out[16]:
[1.1334755420684814, 0.7641893029212952]

Test results are not much different from the validation results. The model performs at about 77% accuracy.

Save the model

In [17]:
# Save the model as a Keras model
save_filename = "image_classifier"
save_filepath = './{}.h5'.format(save_filename)

model.save(save_filepath)

Load the model

In [18]:
model = tf.keras.models.load_model(
    save_filepath,
    custom_objects = {'KerasLayer' : hub.KerasLayer} # model uses KerasLayer which is a part of hub.KerasLayer
)

model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
keras_layer (KerasLayer)     (None, 1280)              2257984
_________________________________________________________________
dense (Dense)                (None, 102)               130662
=================================================================
Total params: 2,388,646
Trainable params: 130,662
Non-trainable params: 2,257,984
_________________________________________________________________

Inference for classification

Now the trained network is going to be used for prediction. With this in mind, predict function will be created. It will take an image, a model, k value and will return top k most likely classes with corresponding probabilities.

Image pre-processing

Before being processed by the model, an image needs to be resized to 224x224 and its colors range needs to be normalized.

In [19]:
# Create the process_image function
# Resize image to 224x224 and normalize the color range
def process_image(image):
    image = tf.cast(image, tf.float32)
    image = tf.image.resize(image, (IMG_SHAPE, IMG_SHAPE))
    image /= 255
    return image

Use a test image to see how it works.

In [20]:
from PIL import Image

image_path = './test-images/hard-leaved_pocket_orchid.jpg'
im = Image.open(image_path)
test_image = np.asarray(im)

processed_test_image = process_image(test_image)

fig, (ax1, ax2) = plt.subplots(figsize=(10,10), ncols=2)
ax1.imshow(test_image)
ax1.set_title('Original Image')
ax2.imshow(processed_test_image)
ax2.set_title('Processed Image')
plt.tight_layout()
plt.show()

Inference

In [21]:
# Create the predict function
def predict(image_path, model, top_k):
    # Prepare image
    image = Image.open(image_path)
    image = np.asarray(image)
    image = process_image(image)
    # Add extra dimension expected by the model
    image = np.expand_dims(image, axis = 0) # Change shape from (224, 224, 3) to (1, 224, 224, 3)

    # Predict
    predictions = model.predict(image)[0]

    # Get indexes of top k predictions
    sorted_indexes = np.argsort(predictions)[::-1] # Sort indexes, in reverse (descending) order
    top_k_indexes = sorted_indexes[0:top_k]

    # Extract probabilites and classes based on indexes
    probabilities = [predictions[i] for i in top_k_indexes]
    classes = [class_names[str(i + 1)] for i in top_k_indexes] # +1 because class_names key range is from 1 to 102
    return probabilities, classes

Prediction

Let's use the model and see how it performs. There are 4 sample images to classify:

  • cautleya_spicata.jpg
  • hard-leaved_pocket_orchid.jpg
  • orange_dahlia.jpg
  • wild_pansy.jpg

Plot the model's predictions against each image and its true class.

In [22]:
# Function plotting image and its true class against model's predictions
def display_prediction(image_path, true_class, model, top_k):
    image = Image.open(image_path)
    image = np.asarray(image)

    probabilities, classes = predict(image_path, model, top_k)

    fig, (ax1, ax2) = plt.subplots(figsize = (10, 10), ncols = 2)
    ax1.imshow(image)
    ax1.set_title(true_class)
    ax1.axis('off')
    y_ticks = np.arange(top_k)
    ax2.barh(y_ticks, probabilities)
    ax2.set_aspect(0.1)
    ax2.set_yticks(y_ticks)
    ax2.set_yticklabels(classes)
    ax2.set_title('Class Probability')
    ax2.set_xlim(0, 1.1)
    ax2.invert_yaxis()  # labels read top-to-bottom
    plt.tight_layout()

# Test images data
test_image_paths = [
    './test-images/cautleya_spicata.jpg',
    './test-images/hard-leaved_pocket_orchid.jpg',
    './test-images/orange_dahlia.jpg',
    './test-images/wild_pansy.jpg'
]

test_image_true_classes = [
    'cautleya spicata',
    'hard-leaved pocket orchid',
    'orange dahlia',
    'wild pansy'
]

# Plot results for each image
for (image_path, true_class) in zip(test_image_paths, test_image_true_classes):
    display_prediction(image_path, true_class, model, 5)

The results for sample images are promising. The model was correct each time. It was certain about two images (hard-leaved pocket orchid and wild pansy), quite confident about one image (cautleya spicata), but it barely recognized the flower type on the fourth image (orange dahlia).

Generate report

In [23]:
!!jupyter nbconvert *.ipynb
Out[23]:
['[NbConvertApp] Converting notebook image_classifier.ipynb to html',
 '[NbConvertApp] Writing 932890 bytes to image_classifier.html']